On the Empirical Evaluation of Las Vegas Algorithms — Position Paper

We advocate a new methodology for empirically analysing the behaviour of Las Vegas Algorithms, a large class of probabilistic algorithms comprising prominent methods such as local search algorithms for SAT and CSPs, like WalkSAT and the Min-Conflicts Heuristic, as well as more general metaheuristics like Genetic Algorithms, Simulated Annealing, Iterated Local Search, and Ant Colony Optimization. Our method is based on measuring and analysing run-time distributions (RTDs) for individual problem instances. We discuss this empirical methodology and its application to Las Vegas Algorithms for various problem domains. Our experience so far strongly suggests that using this approach for studying the behaviour of Las Vegas Algorithms can provide a basis for improving the understanding of these algorithms and thus facilitate further successes in their development and application. 1 Las Vegas Algorithms Las Vegas Algorithms (LVAs) are nondeterministic algorithms for which, if a solution is found, its correctness is guaranteed. However, it is not guaranteed that for soluble instance of decision problems such an algorithm eventually finds a solution, or, for optimisation poblems, that an optimal or close-to-optimal solution will eventually be reached. Because of its nondeterministic nature, the run-time of a Las Vegas Algorithm is a random variable. Las Vegas Algorithms are prominent not only in the field of Artificial Intelligence but also in other areas of computer science and Operations Research. Because of their inherent randomness, stochastic local search (SLS) algorithms are particular instances of LVAs. In recent years, SLS algorithms have become quite prominent for solving both NPcomplete decision problems and NP-hard combinatorial optimisation problems. These 1Corresponding author; address: Computer Science Department, University of British Columbia, 2366 Main Mall, Vancouver, BC, Canada V6T 1Z4 algorithms, such as specific SLS algorithms for SAT and CSPs like WalkSAT [16] and the Min-Conflicts Heuristic [12], respectively, as well as more general metaheuristics like Tabu Search [2], Simulated Annealing [11], Genetic Algorithms [3], Evolution Strategies [14, 15], Ant Colony Optimisation [1], or Iterated Local Search [13, 18] have been found to be very successful on numerous problems from a broad range of domains. But also a number of systematic search methods, like some modern variants of the Davis Putnam algorithm for propositional satisfiability (SAT) problems, make use of non-deterministic decisions (like randomised tie-breaking rules) and can thus be characterised as Las Vegas Algorithms. However, due to their stochastic nature, in analysing the behaviour of Las Vegas Algorithms one is mainly restricted to empirical methods, as theoretical results are difficult to obtain and often very limited in their practical applicability. The latter is, for example, the case for Simulated Annealing, which is proven to converge towards an optimal solution under certain conditions which, however, cannot be met in practice. On the other hand, theoretical results for algorithms which could be shown to be very effective in practice are usually very limited, as is the case for the most successful variants of Tabu Search. Often, the empirical methods that have been applied for the analysis of Las Vegas (and particularly SLS) algorithms in AI have been rather simplistic, like measuring and comparing average or median run-times over a test-set of instances sampled from a random problem distribution such as Random-3-SAT, Random Binary CSP, etc. It can be shown that these methods not only often give a very coarse description of algorithmic behaviour, but that they can also lead to misinterpretations and erroneous conclusions, e.g., when different sources of randomness (such as stochastic choices within the algorithm and probabilistic generation in the instance generation procedure, resp.) are not analytically separated [7]. 2 Empirical Analysis using RTDs We argue that a more sophisticated methodology for the empirical analysis of Las Vegas Algorithms is needed as a basis for their application, investigation, and further development; over the past three years, we developed such a refined empirical methodology and applied it extensively to Las Vegas Algorithms for various problem domains, foremost the well-known satisfiability problem in propositional logic (SAT). Our method is based on measuring and analysing run-time distributions (RTDs) for individual problem instances. For decision problems, this is done by running the given algorithm on the same problem instance for a number of times, where in each of these runs the time for finding a solution is recorded. From this data, an empirical run-time distribution can easily be estimated [7]. (For examples of empirical run-time distributions, see Figure 1.) For optimisation problems additionally the solution quality has to be taken into account. In this case, we measure qualitative run-time distributions for different bounds on the required solution quality (which can be given, for example, as the percentage deviation from the best known solution or a lower bound on the optimal solution value). This can be effectively done by running the optimisation algorithm a number of times on the same problem instance; in each of these runs whenever a new best solution is found, the solution quality, the computation time needed to obtain it, and possibly some other statistic data for further analysis are recorded. This data is sufficient for estimating empirical run-time distributions for different solution quality bounds [18, 4]. (For examples of qualitative run-time distributions, see Figure 2.) Based on our own empirical experience and more general considerations, we suggest the following guidelines for empirically analysing the run-time behaviour of Las Vegas algorithms. 1. Generally measure and compare RTDs, as opposed to basic descriptive statistics like mean, standard deviation, or percentiles. Enough runs of the algorithm should be performed to ensure that the estimates for the RTDs are sufficiently stable. 2. Try to approximate the empirical RTDs using parameterised functional models. Statistical goodness-of-fit tests like the 2-test should be used to evaluate these functional models. In our studies of SLS algorithms for various problem domains, we found that often approximations using exponential or generalised exponential distributions [4] were surprisingly accurate. 3. When comparing algorithms, check for cross-overs in their RTDs. If present, these typically indicate that by using portfolios or hybrid algorithms the problem can be solved more robustly and/or more efficiently. 4. When dealing with parameterised algorithms, it is often desirable and rewarding to study the impact of these parameters on the algorithm’s RTD. When multiple parameters are used, care should be taken to avoid premature assumptions on the independence of their effects. 5. When using sets of problem instances (randomly generated or not), RTDs should be measured for individual instances to clearly detect differences in run-time behaviour across the test-set. Ideally, if the individual RTDs can be characterised using a parameterised model, the distribution of the model parameters across the test-set should be studied. If a parameterised model is not available, the distribution of basic descriptive statistics of the individual RTDs should be investigated. For sets of randomly generated instances this method ensures that the different sources of randomness (in the problem generation procedure and the LVA to be evaluated) are clearly separated. 6. Likewise, when comparing the performance of different algorithms on sets of instances, the comparison should be made on an individual instance basis. This allows to precisely analyse the correlation between the algortithms’ performance across test-sets using standard statistical techniques. Note that while we advocate to characterise the observed RTDs using parameterised functional models, many aspects of the RTD-based methodology do not rely on this so that they are still applicable when such models are not available. The advantage of parameterised approximations is twofold: first, they allow the compact and yet accurate characterisation of the observed run-time behaviour and secondly, by they often suggest generalisations and deeper explanations of the observed behaviour, in particular if the model parameters can be linked to algorithm parameters or properties of the given problem instances. 3 Applications of an RTD-based Methodology Based on a classification of application scenarios for Las Vegas Algorithms, we have shown that in general, only RTDs provide all the information required to adequately describe the behaviour of these algorithms [4, 7]. We also demonstrated, how, based on functional approximations of RTDs for individual problem instances, interesting and novel characterisations for the run-time behaviour of some of the most popular stochastic local search algorithms in various areas of AI can be obtained, including SAT, Constraint Satisfaction, and various combinatorial optimisation problems [8, 10, 4, 18]. These characterisation results are of both, qualitative and quantitative nature and have a number of practical as well as theoretical implications. The RTD-based methodology also provides a good basis for adequately comparing the performance of different Las Vegas Algorithms. Here, we additionally advocate the use of benchmark libraries comprising fundamentally different types of problems: instances from random problem distributions, such as Random-3-SAT; individual, application-relevant or otherwise interesting instances; and, where applicable, randomly generated, encoded instances from other domains (such as SAT-encoded graph colouring problems in random graphs). The latter have the advantage that they combine aspects of randomly generated and structured problem instances and are therefore, in our opinion, ideally suited for studying the impact of certain structural aspects of the given problem instances on algorithmic performance [6]. Based on these principles, we created and maintain SATLIB2, a comprehensive public repository of SAT problem instances and algorithms; using our RTD-based methodology, we also conducted a large-scale empirical study comparing the 2www.informatik.tu-darmstadt.de/AI/SATLIB 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10 10